现代高性能计算面临一个根本性挑战 “内存墙”：计算吞吐量（每秒浮点运算次数，FLOPS）的爆炸式增长，远远超过了内存带宽的缓慢提升 全局内存 带宽。这种差异导致大规模多核阵列变成‘饥饿’的处理器，只能等待数据到达。
1. 带宽差距
尽管GPU每秒可执行数万亿次操作，但通往DRAM的物理路径受限于引脚密度和功耗要求。 内存作为并行性的限制因素 意味着随着线程数量的增加，每个线程的带宽下降，从而导致硬件处于空闲等待状态的停顿周期。
2. 厨房类比
想象一个现代化的厨房（即GPU核心），每小时能烹饪1000份餐食。然而，食材存放在五英里外的仓库中（即全局内存），而运送工具只有一辆快递摩托车（即内存总线）。无论你雇佣多少厨师，你的产出都受限于这辆摩托车的速度。
3. 架构对比
标准的 多核CPU系统 利用巨大的缓存来隐藏少数重型线程的延迟。然而，大规模并行架构却持续面临并发请求的“交通堵塞”。 资源限制 在寄存器和共享内存层级上的资源限制，决定了硬件被压垮前所能达到的最大并行度（占用率）。
main.py
TERMINALbash — 80x24
> Ready. Click "Run" to execute.
>
QUESTION 1
What is the primary cause of the 'Memory Wall' in modern GPU computing?
The clock speed of cores is too slow to process DRAM data.
Computational throughput (FLOPS) has increased much faster than memory bandwidth.
Shared memory is too large for the hardware to manage.
Global memory has higher latency than CPU registers.
✅ Correct!
This gap creates a bottleneck where processors spend most of their time waiting for data delivery.❌ Incorrect
The issue is the growth rate discrepancy between processing speed and the physical bandwidth of the memory bus.QUESTION 2
In the 'Kitchen Analogy,' what does the delivery scooter represent?
The GPU Core/Chef.
The Register File.
The Global Memory Bus.
The Operating System Scheduler.
✅ Correct!
The bus (scooter) is the narrow pipe that limits how fast 'ingredients' (data) reach the 'chefs' (cores).❌ Incorrect
The scooter represents the transmission medium, not the compute resource or the storage itself.QUESTION 3
How do resource limitations like register count affect parallelism?
They increase the speed of each individual thread.
They limit occupancy by reducing the number of active threads that can reside on an SM.
They have no effect on throughput, only on power consumption.
They bypass the need for global memory access.
✅ Correct!
Since hardware has a fixed pool of registers, using more registers per thread forces the GPU to run fewer concurrent threads.❌ Incorrect
Exceeding per-thread resource limits directly lowers 'occupancy,' meaning fewer threads are available to hide memory latency.QUESTION 4
When a kernel is in the 'Memory Bound' region of the Roofline Model, what is the best way to improve performance?
Increase the number of floating-point operations per second.
Increase the arithmetic intensity (data reuse).
Decrease the number of threads per block.
Add more complex branching logic.
✅ Correct!
Increasing arithmetic intensity (reusing data from shared memory) moves the kernel closer to the compute-bound plateau.❌ Incorrect
In a memory-bound state, adding more math won't help if the bottleneck is fetching data from memory.QUESTION 5
Why is implicit synchronization unreliable in massively parallel architectures?
Hardware evolution means threads within a warp may not stay locked in SIMT fashion.
Shared memory is too fast for synchronization to matter.
Global memory access is always synchronous.
Threads are processed sequentially in blocks.
✅ Correct!
Always use `__syncthreads()` to ensure data consistency, as hardware execution order is not guaranteed.❌ Incorrect
Relying on warp-level timing is dangerous; explicit barriers are mandatory for correctness in shared memory access.Case Study: Memory Optimization Audit
Analyzing Matrix Operations
You are auditing two kernels: Kernel A performs simple Matrix Addition ($C = A + B$). Kernel B performs Matrix Multiplication ($C = A \times B$). You apply Shared Memory Tiling to both.
Q
1. Which kernel will see a significant reduction in global memory bandwidth consumption after tiling?
Solution:
Kernel B (Matrix Multiplication). In multiplication, each element is used multiple times by different threads, allowing reuse via tiling. In addition, each element is accessed exactly once by one thread, so tiling offers no reuse benefit.
Kernel B (Matrix Multiplication). In multiplication, each element is used multiple times by different threads, allowing reuse via tiling. In addition, each element is accessed exactly once by one thread, so tiling offers no reuse benefit.
Q
2. If an SM has 8,192 registers and a thread limit of 768, what is the maximum registers a thread can use to maintain 100% occupancy?
Solution:
$8,192 / 768 \approx 10$ registers per thread. If a kernel uses 11 registers, the occupancy will drop because the SM cannot fit all 768 threads simultaneously.
$8,192 / 768 \approx 10$ registers per thread. If a kernel uses 11 registers, the occupancy will drop because the SM cannot fit all 768 threads simultaneously.
Q
3. Explain the risk of a Read-After-Write (RAW) hazard if `__syncthreads()` is omitted after loading a tile.
Solution:
Without the barrier, a thread might attempt to perform a calculation using a value in shared memory before the thread responsible for loading that specific value has actually finished writing it from global memory.
Without the barrier, a thread might attempt to perform a calculation using a value in shared memory before the thread responsible for loading that specific value has actually finished writing it from global memory.